Skip to content

repack: the container is verified against the GGUF it came from, under the name that says so (refs #1109) - #1234

Merged
bong-water-water-bong merged 6 commits into
mainfrom
1bit/repack-verify-source
Nov 6, 2026
Merged

bong-water-water-bong merged 6 commits into
mainfrom
1bit/repack-verify-source

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Refs #1109. Follows the issue's own last comment: the guard is "verify the container it just wrote against the GGUF it came from", not "name a table of converter commits".

What was already there (checked before writing anything)

#1169's tile check already compares the container against its source, not just against layouts: tools/q4nx_diff.py:332 dequantizes the GGUF tensor (ref = dequantize(t.data, t.tensor_type)) and :333 compares every decoded container tensor with it, under both layouts, called from repack_gguf.py after the converter returns. So this PR adds no second comparison path. Measured on the issue: a wrong converter checkout reads matches NO layout with corr +0.0325 where an honest pack reads corr +0.9994.

What was missing, and is here

  • --verify-source <dir> <model.gguf> and the reusable verify_source() it wraps (tile_order_check kept as the old name): the measurement under the name the issue asks for, usable on an existing or published directory. Exit 0 verified in the engine-1 order, 1 another order or none, 2 nothing comparable. The repack calls the same function, so the two cannot disagree.
  • The verdict names the source tensor and its correlation: matches NO layout: closest is flm with worst max|err| 3.460e+00, corr +0.0325 on blk.14.attn_output.weight. Reported, not gated — no correlation floor was measured on honest containers, and the gate stays the measured max|err| rule so a guess cannot refuse a good pack.
  • The refusal names the converter checkout @ commit (repack: converter: <dir> @ <commit>), the one fact repack-stamp.txt has and the person hitting it needs.
  • --skip-verify is the escape the issue's comment asks for (the verify costs a GGUF pass, which matters in the tight repack loops the payload builds run). It stamps tile_order=unverified with tile_check=skipped (--skip-verify) — never engine-1.
  • A reuse hole closed. The refusal path leaves model.q4nx on disk, and --check-stamp ignored refused/tile_order: the next serve reported stamp: current and served the container the repack had just rejected. Now refused→stale (1), a stamp that predates the measured tile_order→stale (rebuilt once, so it becomes verified), unverified→2 ("cannot tell": serve reuses it with its warning, and does not re-run the measurement that could not run).

Tests

python3 tests/repack_verify_source_test.py — new, 0 failures, wired into ci.yml's python step (it needs numpy + gguf, which that step installs). It drives the real CLI with a one-tensor synthetic GGUF, one-tile containers packed in each layout, and a fake converter: good container matches with corr +1.0000 and the tensor named; other values match nothing with corr near 0; an FLM-order container matches its source but is refused as this engine's order; --verify-source exits 0/1/2; --skip-verify stamps unverified and --check-stamp returns 2 for it; a real repack stamps tile_order=engine-1 with the measured verdict; a refused directory is stale on the next cache hit.

Also green: tools/copyright.py --check, q4nx_diff.py --self-test, tests/q4nx_diff_grid_test.py, tests/repack_tile_order_test.py, tests/tile_check_no_shared_tensor_test.py, tests/npu_plan_tile_order_test.py, tests/gguf_to_q4nx_test.py.

Not verified

No real GGUF/converter/model and no end-to-end serve — this box is device-free for that path, as scoped. The synthetic arms exercise the same entry points.

Reviewer notes

  1. Product call: a stamp written before repack: a container's q4_1 tile order is measured and a wrong one is refused (#1110) #1169 has no measured tile_order, so --check-stamp now returns 1 and serve rebuilds that cache dir once. A rebuild is the only way such a dir becomes verified. It only happens when the converter resolves (otherwise the answer is 2, fail open), so a box without its converter is not lost — but it is a one-time cost per cached dir on upgrade. Say if fail-open (2) is preferred for pre-measurement stamps.
  2. docs/npu.md (Lead-added in the second commit) now documents --verify-source, --skip-verify and the rebuild-once rule.
  3. Deliberately not done: attention-only scoping (padded MLP tensors are readable since q4nx-diff: read a container whose weight is padded to a wider tile grid (part of #1110) #1119/dd5fe38, so scoping would weaken the guard), and a correlation threshold.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

PR Reviewer Guide 🔍

(Review updated until commit 5a8f5ab)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1109 - Partially compliant

Compliant requirements:

  • Fail loud on an unknown stamp at serve time
  • Verify the repack against its source

Non-compliant requirements:

  • Name the validated checkout in docs/npu.md and in the resolver's candidate list

Requires further human verification:

  • The documentation update in docs/npu.md needs to be verified for completeness and accuracy

1169 - Partially compliant

Compliant requirements:

  • The repack now verifies the container against its source GGUF
  • The repack refuses containers with incorrect tile order

Non-compliant requirements:

  • None

Requires further human verification:

  • None

1119 - Partially compliant

Compliant requirements:

  • The grid is derived from the tensor's data_size exactly as npu/forward's q4nx_packed_grid() does
  • The padded tensor is decoded at the container's grid and only the model's [rows, cols] slice is compared
  • The summary gains padded=N

Non-compliant requirements:

  • None

Requires further human verification:

  • None
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Incorrect Error Handling in Tile Order Check

The tile_check_exit function in scripts/repack_gguf.py incorrectly returns exit code 1 when the tile order is fatal, but it should return 0 for success and 1 for failure. This can cause the repack to incorrectly refuse directories when the tile order is valid.

def tile_check_exit(rc, line):
    """Exit code for `--tile-check`: 0 ours, 1 another order or none, 2 nothing compared.

    The checker's own rc is not this: `verdict()` returns 0 for ANY matching layout, including the
    one this engine does not decode, so the CLI goes through the policy above instead of passing it
    through (engine#1110 review, blocker 1).
    """
    order, fatal = tile_order_policy(rc, line)
    if fatal:
        return 1
    return 2 if order == "unverified" else 0
Incorrect Usage of `--skip-verify` Flag

The --skip-verify flag in scripts/repack_gguf.py is not properly handled in the main function, which can lead to incorrect behavior when skipping verification.

def main():
    # `--skip-verify` is a flag on the repack form, not a subcommand: the verification decodes the whole
    # container against the GGUF and costs a GGUF pass, so the tight repack loops the payload builds run
    # in can opt out (engine#1109).  Opting out is recorded in the stamp as unverified, never as engine-1.
    skip_verify = "--skip-verify" in sys.argv[1:]
    argv = [a for a in sys.argv[1:] if a != "--skip-verify"]
    if len(argv) < 2:
        print("usage: repack_gguf.py <model.gguf> <output_dir> [converter_dir] [--skip-verify]")
        print("       repack_gguf.py --check-stamp <dir>")
        print("       repack_gguf.py --verify-source <dir> <model.gguf>")
        print("       repack_gguf.py --tile-check <dir> <model.gguf>")
Correlation Calculation in Verdict Function

The verdict function in tools/q4nx_diff.py calculates the correlation between decoded weights and source weights, but it does not properly handle cases where the correlation is NaN, which can lead to incorrect verdicts.

def verdict(results):
    """(rc, line) for {layout: (exact, inexact, worst_max_err, worst_tensor, corr_at_worst)}.

    A container is laid out ONE way, and a caller that guesses wrong gets a confident "BAD" for a good
    container. So name the layout that matched, and when none did, name the closest and say so instead
    of implying the container is broken.  The line names the worst tensor and its correlation with the
    source as well (engine#1109), so "the wrong converter checkout" reads as
    `corr +0.0325 on blk.14.attn_output.weight` rather than as a bare error.
    """
    # A layout that compared NO tensor is not evidence of anything: `_decode_pass` returns
    # (0, 0, 0.0, "", nan) when every tensor was skipped, and worst=0.0 with rival=0.0 satisfies
    # "rival >= worst * RATIO", so such a layout used to be reported as a MATCH on whichever one
    # sorted first -- measured: a 0.6B container against a 4B GGUF came back
    # `matches layout flm (worst max|err| 0.000e+00, ...)` with 0 tensors compared (engine#1110
    # review).  Drop the layouts with no evidence before choosing, and if none is left say so:
    # every caller already has a branch for rc=2.
    results = {k: v for k, v in results.items() if v[0] + v[1] > 0}
    if not results:
        return 2, "nothing was compared"
    order = sorted(results, key=lambda k: results[k][2])
    best = order[0]
    exact, inexact, worst, worst_name, worst_corr = results[best]
    rival = results[order[1]][2] if len(order) > 1 else None
    took = (exact > 0 and worst == 0.0) or (
        worst <= LAYOUT_MATCH_MAX_ERR and (rival is None or rival >= worst * LAYOUT_MATCH_RATIO))
    # Name the tensor and its correlation with the source in both verdicts: a wrong converter checkout
    # is exactly "same geometry, permuted values", and corr +0.03 vs +0.999 is what tells a reader
    # which side of that line a container is on (engine#1109).
    corr = "corr %s" % _corr_text(worst_corr)
    on = (" on " + worst_name) if worst_name else ""
    if took:
        other = "" if rival is None else ", next layout %s worst max|err| %.3e" % (order[1], rival)
        how = ("%d tensor(s) exactly, %s%s" % (exact, corr, on) if worst == 0.0 and exact
               else "worst max|err| %.3e, %s%s" % (worst, corr, on))
        return 0, "matches layout %s (%s%s)" % (best, how, other)
    return 1, ("matches NO layout: closest is %s with worst max|err| %.3e, %s%s (> %.3e, or not %.0fx "
               "better than %s) - the container's tiles are not these weights"
               % (best, worst, corr, on, LAYOUT_MATCH_MAX_ERR, LAYOUT_MATCH_RATIO,
                  order[1] if rival is not None else "-"))

bong-water-water-bong added 2 commits November 5, 2026 23:46
…against a converter's name (#1109)

The comparison this issue asks for already landed inside #1169's tile check: `tile_order_check` runs
`q4nx_diff._decode_pass`, which dequantizes the source GGUF (`ref = dequantize(t.data, t.tensor_type)`)
and compares every decoded container tensor against it under BOTH layouts.  So a wrong checkout is
already refused (`matches NO layout`, rc=1) -- what was missing is the contract around it, and the
hole that reopened after the refusal.

  * `--verify-source <dir> <model.gguf>` (and the reusable `verify_source()` it wraps, with
    `tile_order_check` kept as its old name): the measurement under the name #1109 asks for, usable on
    an existing or published directory.  Exit 0 verified in the engine-1 order, 1 another order or
    none, 2 nothing comparable.  The repack calls the same function after the converter and the
    config write, so the two cannot disagree.
  * the verdict now names the SOURCE tensor and its correlation: a wrong checkout reads
    `matches NO layout: closest is flm with worst max|err| 3.460e+00, corr +0.0325 on
    blk.14.attn_output.weight`, against an honest pack's `corr +0.9994` (the issue's numbers).  The
    correlation is reported, not gated: no floor was measured on honest containers, and the gate
    stays the measured max|err| rule.
  * the refusal names the converter checkout and commit the stamp is about to record -- the one fact
    repack-stamp.txt has and the user needs when a converter is the wrong version.
  * `--skip-verify` is the escape the issue's comment asks for (the verify costs a GGUF pass, for the
    tight loops the payload builds run in).  It stamps `tile_order=unverified` with
    `tile_check=skipped (--skip-verify)` -- never `engine-1`.
  * `--check-stamp` no longer says `stamp: current` for a directory the repack REFUSED: the refusing
    path leaves model.q4nx on disk, so before this the next serve reused the container the guard had
    just rejected -- a wrong-converter container still served after being caught.  A stamp that
    predates the measurement is stale (rebuilt, which is what makes it verified); a stamp that says
    `unverified` is "cannot tell" (2), so serve reuses it with its existing warning instead of
    silently, and does not re-run the failed measurement on every start.

Deliberately NOT added: a second comparison path (the existing one compares against the source, not
just against layouts), a correlation threshold (not measured on honest containers; a guess would
refuse them), and attention-only scoping (padded MLP tensors have been readable since #1119/dd5fe38,
so scoping would weaken the check the comment asked for).

tests/repack_verify_source_test.py: a one-tensor synthetic GGUF, a one-tile container packed in each
layout, and a fake converter that copies a prepared container, through the real entry points.  It
asserts the good container matches with corr +1.0000 and the tensor named, other values match nothing
with corr near 0, an FLM-order container matches its source but is refused as this engine's order,
`--verify-source` exits 0/1/2, `--skip-verify` stamps unverified and `--check-stamp` returns 2 for it,
a real repack stamps `tile_order=engine-1` with the measured verdict, and a refused directory is stale
on the next cache hit.  Wired into ci.yml's python test step (numpy + gguf installed there).

Run: copyright rc 0; the test above 0 failures; q4nx_diff --self-test, q4nx_diff_grid,
repack_tile_order, tile_check_no_shared_tensor, npu_plan_tile_order, gguf_to_q4nx all pass.
…eds, and a pre-measurement stamp is rebuilt once (#1109)
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

#1234 ready for review at 64050aa

Rebased onto (the branch was BEHIND), no content change beyond the rebase. was green on the previous head (, 6m50s); new run is on this head.

Verified on this branch, device-free: (25 assertions, wired into ci.yml's python step), , , , , , , .

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit 64050aa

@bong-water-water-bong
bong-water-water-bong enabled auto-merge (squash) November 6, 2026 03:53
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Head moved to via GitHub's update branch (main merged in, no content change): the branch had gone BEHIND within one build cycle. was green on (5m30s). Auto-merge is armed; the queue can land it as soon as this head's is green.

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit e76d04f

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

Persistent review updated to latest commit 5a8f5ab

@bong-water-water-bong
bong-water-water-bong merged commit aed110c into main Nov 6, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the 1bit/repack-verify-source branch November 6, 2026 04:18
bong-water-water-bong added a commit that referenced this pull request Nov 7, 2026
…a foreign-order directory is refused offline (Fixes #1110) (#1275)

* npu: check_q4nx_dir --against measures the q4_1 TILE ORDER, so a foreign-order dir is refused offline

engine#1110's surviving hole: the repack measures the tile order (#1169/#1234) and the plan
loader refuses a declared order it does not implement (#1130), but a directory that was never
repacked here -- imported from a foreign converter, downloaded, copied off a board -- has no
stamp and no plan, and every metadata check passes: the two orders share the chunk kind, the
byte count and the tensor names.  Only the weights can say which order the bytes are in.

--against already decoded the first Q8 tile both ways against the source GGUF; it now does the
same for q4_1 over all three orders this repo can name -- ours (npu/q4nx.cpp's), flm signed,
flm unsigned -- with tools/q4nx_diff.py's own decoders imported rather than re-implemented, and
refuses a container whose winner is not ours: tensor name, all three correlations, and the
closest foreign reading in the message.  Accepted when ours >= 0.99 and ours - best_other >=
0.5, both measured: a container packed from Q4_K_M reads +1.0000, one packed from BF16 +0.996,
every foreign order |corr| < 0.05.  The metadata-only path still imports no numpy and no gguf.

Tests: tests/check_q4nx_tile_order_test.py builds the case the issue describes -- one known
32x256 weight encoded as a single q4_1 tile in the engine's order and again in the foreign one,
each encoder round-tripping through its own decoder first (4.17e-03) while the same bytes read
as 'ours' are 1.31e-01 wrong -- then requires pass/refuse, the refusal to name the tensor, the
foreign reading to be identified, and the verdict to survive a rename to another tensor.

* npu: decide the q4_1 order on the margin, not a 0.99 proximity cut, and make the self-test follow the message

Review blockers, both real.

1. `--self-test` exited rc 1 at every commit of this branch: the summary line was reworded
   ("no Q8 or q4_1 tile was decoded ...") and the assertion at the old line 522 still looked for
   "no Q8 tile was decoded".  My own run had printed that failure and I quoted "self-test ok" from
   its tail without reading its exit code -- the exact claim-before-check mistake this lane keeps
   writing down.  Fixed, and the self-test now also covers the q4_1 selector and the sampler
   (pure python: a 5120-B I8 tensor is q4_1, 4736/8704 are not; spread() reaches both ends).

2. `ours >= 0.99` was a proximity cut on quantization error, and it refused the PR's own
   non-regression row: on a full sweep of the published FastFlowLM Qwen3-0.6B-NPU2 directory the
   outlier-heavy `layers.2.mlp.down_proj` reads corr +0.9890 (rel-rms 0.1477) with a margin of
   ~0.95 over every foreign reading -- unmistakably the engine's order -- yet the check refused it
   and then said "NOT in the order, best is ours", which contradicts itself.

   The order is now decided on the MARGIN -- ours wins and beats the runner-up by >= 0.5 -- with a
   0.9 floor that only rules out "no reading matches at all".  Measured: correct containers lead by
   >= 0.95, foreign ones by <= 0.05 (the d1d5232 container: ours +0.0477 vs flm-unsigned +0.9961).
   The refusal now says which of the three things happened instead of one message for all:
   no reading matches (wrong GGUF or an unnameable order), a foreign order it can name, or an
   ambiguous margin (a partly foreign container).

Also, per the review's non-blocking note: samples are spread evenly across the table instead of the
first N in table order, so a foreign order that varies tensor to tensor is not missed by sampling one
layer.

Tests: tests/check_q4nx_tile_order_test.py grows to 14 checks.  The new ones are the regression the
floor caused (a correct container at corr +0.9827 must pass -- inside the band the old floor refused)
and the wrong-source case (must say "no reading of these tiles matches", never claim a foreign order
it cannot identify).  Negative control: restoring `ours >= 0.99` makes exactly the new regression
check fail and nothing else, so the test bites on this bug and not on a coincidence.

---------

Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant